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SUBJECT:  Audit  Report  on  the  Acquisition  and  Utilization  of 

Modeling  and  Simulation  Assets  at  Contractor 
Facilities  (Project  No.  3AB-0009) 


introduction 

We  are  providing  this  final  report  for  your  information 
and  use.  Expanding  technological  capability  has  enabled  the 
application  of  computer-assisted  analysis  to  more  diverse 
fields  of  study  in  an  increasingly  sophisticated  manner. 
This  capability  has  resulted  in  a  significant  increase  in 
the  number  and  frequency  of  use  of  modeling  and  simulation. 
The  primary  objective  of  this  audit  was  to  determine  whether 
DoD  is  funding  redundant  modeling  and  simulation  projects 
for  Defense  contractors. 

A  "model"  is  a  physical,  mathematical,  or  other  logical 
representation  of  a  system,  entity,  phenomenon,  or  process. 
A  "simulation"  is  a  method  for  implementing  a  model  over 
time,  as  well  as  a  technique  for  testing,  analyzing,  or 
training  in  which  real  world  and  conceptual  systems  are 
reproduced  by  a  model.  "Simulation"  is  also  defined  as  a 
model  of  a  "real  world"  situation.  The  terms  "model"  and 
"simulation"  are  often  used  interchangeably.  Design  of  a 
model  starts  with  assumptions  representing  this  "real  world" 
situation  as  a  mathematical  model  (equations) ,  list  of 
events,  or  a  combination  of  equations  and  events. 

Defense  contractors  use  models  and  simulations  to 
mathematically  represent  and  test  alternatives  and  concepts 
when  designing  a  weapon  system.  Models  and  simulations  are 
used  for  analytical  purposes  to  dynamically  represent  a 
conceptual  weapon  system  during  research,  development,  and 
test  and  evaluation  of  weapon  programs  undertaken  for  DoD. 
This  audit  focused  on  Defense  industry  investments  in 
modeling  and  simulation.  These  efforts  are  funded  primarily 
by  DoD  through  IR&D  or  general  and  administrative  overhead 
items  of  expense  on  negotiated  contracts. 


I 


2 


The  audit  is  a  continuation  of  a  requested  audit.  In 
October  1990,  the  Deputy  Director,  DoD  Office  of  Acquisition 
Policy  and  Program  Integration,  requested  that  the  Inspector 
General,  DoD,  study  possible  duplication  and  proliferation 
of  modeling  and  simulation  efforts  in  DoD.  In  response,  we 
performed  the  "Audit  of  Duplication/Proliferation  of  Weapon 
Systems  Modeling  and  Simulation  Efforts  within  DoD, 
Report  No.  93-060,  March  1,  1993.  The  scope  of  this 

audit  was  limited  to  DoD-owned  models  and  simulations  and 
disclosed  three  issues  requiring  further  management 
attention.  DoD  management  concurred  with  the  findings  and 
recommendations  and  has  initiated  corrective  action. 

o  •  Model  and  simulation  projects  are  being  procured  and 
developed  within  the  DoD  without  adequate  coordination  and 
control  resulting  in  costly  duplication. 

o  The  vast  majority  of  models  and  simulations  currently 
used  in  DoD  have  not  been  verified,  validated,  or 
accredited.  Weapon  system  requirement  decisions, 

development  decisions,  engineering  designs,  operations,  and 
test  and  evaluation  results  may  be  based  on  computer- 
generated  data  that  is  inaccurate  or  misleading. 

o  The  majority  of  models  and  simulations  used  in  DoD 
lack  adequate  configuration  management  and  documentation 
necessary  to  ensure  ready  access  by  authorized  Defense 
personnel. 

Objectives 

This  economy  and  efficiency  audit  was  conducted  in 
accordance  with  auditing  standards  issued  by  the  Comptroller 
General  of  the  United  States  as  implemented  by  the  Inspector 
General,  DoD,  and  accordingly  included  such  tests  of 
internal  controls  as  were  deemed  necessary.  Our  primary 
objective  was  to  determine  whether  DoD  is  funding  redundant 
modeling  and  simulation  projects  for  Government  contractors. 
Accordingly,  our  audit  was  to  determine  whether  redundancy 
exists  between  models  and  simulations  owned  by  DoD  and  those 
funded  by  DoD  for  Government  contractors. 

Scope  of  Audit 

This  audit  was  made  from  October  1992  through  March 
1993.  We  reviewed  a  judgmental  sample  of  20  contractor 
models  and  simulations.  Each  sample  model  was  analyzed  for 
possible  redundancy  and  evidence  of  verification, 
validation,  and  accreditation.  We  also  analyzed  each  sample 
model  for  adequacy  of  system  documentation  and  configuration 
management  plans  and  controls.  The  Quantitative  Methods 
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Division  of  the  IG,  DoD,  assisted  in  the  analysis  of  sample 
models.  Specific  contractor  locations  visited  are  listed  in 
Enclosure  1. 

internal  Controls 

We  evaluated  the  effectiveness  of  internal  controls 
established  to  prevent  redundant  or  unnecessary  investment 
in  models  and  simulations  at  Government  contractor 
facilities.  The  audit  identified  material  internal  control 
weaknesses  as  defined  by  Public  Law  97— 255,  Office  of 
Management  and  Budget  Circular  A-123,  and  DoD  Directive 
5010.38.  However,  the  National  Defense  Authorization  Acts 
for  Fiscal  Years  1991  and  1992  and  for  Fiscal  Years  1992 
and  1993  (Defense  Authorization  Acts)  essentially  negate  the 
DoD' s  authority  to  exercise  reasonable  control  over 
contractors'  Independent  Research  and  Development  (IR&D)  and 
Bid  and  Proposal  (B&P)  expenditures,  the  source  of  funding 
for  most  contractor-developed  models  and  simulations. 
Hopefully,  Government  controls  will  be  replaced  by 
contractor  controls.  Because  DoD  has  limited  ability  to 
address  the  problems  we  identified,  we  have  terminated  the 
project  and  are  issuing  this  report  of  our  observations 
without  recommendations . 

Prior  Audit  Coverage 

In  addition  to  the  Inspector  General,  DoD,  "Audit  of 
Duplication/Proliferation  of  Weapon  Systems  Modeling  and 
Simulation  Efforts  within  DoD,"  Report  No.  93-060,  discussed 
previously,  a  recently  issued  General  Accounting  Office 
audit  report  addressed  Independent  Research  and  Development 
and  Bid  and  Proposal  expenditures. 

General  Accounting  Office  Report  (GAO)  No.  GAO /NS I AD-B- 
207974  (OSD  Case  No.  9138^ .  "Government  Contracting^ 
Proposed  Regulation  Would  Limit  DoD's  Ability  to  Review 
IR&D/B&P  Program."  September  1992.  The  GAO  reported  that 
removing  IR&D/B&P  ceilings  would  increase  costs  to  the 
Government  between  $100  million  and  $1  billion  per  year. 
GAO  also  reported  that  DoD  administrative  costs  are  not 
likely  to  be  significantly  reduced  as  intended  by  the 
proposed  legislation.  The  GAO  recommended  that  the  Defense 
Contract  Audit  Agency  continue  preparing  an  annual  financial 
report  on  IR&D/B&P  expenditures. 

Background 

Over  the  years,  contractors  have  developed  computer 
models  on  an  "ad  hoc"  basis  to  analyze  specific  contract- 
related  problems,  issues,  and  questions.  This  practice  has 
resulted  in  contractor-owned  models  written  in  a  multitude 
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of  computer  codes,  with  different  designs  and  language 
implementation.  When  these  models  are  compared  with 
internal  DoD-owned  models,  they  are  frequently  oriented 
toward  accomplishing  similar,  if  not  identical,  types  of 
analyses. 

This  audit  focused  on  models  and  simulations  developed 
wholly  or  in  part  with  DoD  funding  at  Defense  contractors. 

Audit  Results 

The  same  three  issues  identified  in  our  Audit  Report 
No.  93-060,  "Duplication/Proliferation  of  Weapon  Systems' 
Modeling  and  Simulation  Efforts  Within  DoD,"  were  identified 
as  problems  at  Government  contractor  locations. 

Redundant  Models.  Specifically,  models  and  simula¬ 
tions  developed  by  Government  contractors  were  redundant  to 
existing  models  and  simulations  within  the  DoD.  We  reviewed 
20  contractor  models  and  found  3  (15  percent)  that  were 
redundant  to  models  and  simulations  already  existing  in  DoD 
and  available  for  use  by  the  Defense  contractors.  DoD 
provided  the  funding  for  development  by  the  contractors  of 
these  redundant  models  and  simulations  either  through  IR&D 
or  general  and  administrative  expense  applied  as  overhead 
items  to  DoD  contracts. 

When  doing  our  "Audit  of  Duplication/Proliferation  of 
Weapon  Systems  Modeling  and  Simulation  Efforts  within  DoD," 
Report  No.  93-060,  we  compiled  a  list  of  models  and 
simulations  available  to  the  Defense  community  from  30 
individual  compendiums  and  catalogues.  Eliminating  duplic¬ 
ations  on  lists  obtained  from  different  sources,  we 
identified  more  than  1,900  models  and  simulations  available 
for  direct  or  indirect  support  of  weapon  system  development. 
The  most  comprehensive  list  of  models  that  we  identified  was 
the  "Catalog  of  Wargaming  and  Military  Simulation  Models," 
12th  Edition,  published  by  the  Joint  Staff,  Force  Structure, 
Resource  and  Assessment  Directorate  (J-8) .  This  list 
contains  522  distinct  models  and  simulations. 

During  this  audit,  we  continued  to  collect  lists  of 
models  or  simulations  used  by  each  contractor  visited.  A 
total  of  171  distinct  models  were  identified.  Only  57  of 
these  models  (33  percent)  had  been  previously  identified  in 
our  existing  compilation  assembled  from  the  numerous 
compendiums  and  catalogs  referenced  above.  We  believe  this 
lack  of  visibility  contributes  to  unnecessary  proliferation 
of  models  and  simulations  because  potential  users  have  no 
reliable  way  to  determine  what  models  and  simulations 
already  exist  that  might  satisfy  their  immediate 
requirement. 
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Verification.  Validation,  and  Accreditation.  The 
majority  of  models  and  simulations  currently  used  by  the 
contractors  we  reviewed  had  not  been  formally  verified, 
validated,  or  accredited.  Only  4  of  the  20  (20  percent) 
contractor  models  evaluated  during  our  audit  had  completed  a 
formal  verification,  validation,  and  accreditation  process. 
An  incomplete  or  informal  verification,  validation,  and 
accreditation  had  been  completed  on  10  of  the  20  (50  per¬ 
cent)  sample  models  but  no  attempt  at  verification, 
validation,  or  accreditation  had  been  made  on  6  of  the 
20  (30  percent)  contractor  models.  As  a  result,  DoD  may  be 
using  the  results  of  contractor-prepared  models  and 
simulations,  where  the  reliability  of  the  analyses  has  not 
been  demonstrated,  to  help  define  weapon  system  requirements 
and  conduct  developmental  tests  and  evaluations. 

Configuration  Management  and  Documentation.  Adequate 
configuration  management  and  documentation  was  lacking  for 
models  and  simulations  developed  by  Government  contractors 
we  reviewed.  Of  20  applications  reviewed,  only  one  model 
(5  percent)  had  adequate  configuration  management  control 
procedures  in  place.  Ten  of  the  models  reviewed  (50  per¬ 
cent)  had  some  formal  or  informal  configuration 
management,  but  9  models  reviewed  had  no  essential  elements 
in  place  for  effective  configuration  management. 

We  also  reviewed  all  available  documentation  for  each 
model  and  simulation  in  our  sample.  We  had  previously 
determined  that,  as  a  minimum,  the  documentation  should 
comprise  a  User's  Manual,  Analyst's  Manual(s),  and 
Programmer's  Manual(s).  Accordingly,  of  the  20  models  and 
simulations  surveyed,  only  7  models  (35  percent)  had 
adequate  documentation.  For  the  remaining  13  models,  the 
documentation  was  either  inadequate  or  did  not  exist. 

Overall,  our  findings  indicate  a  substantial  lack  of 
effective  management  control  over  the  development  and  use  of 
models  and  simulations  by  the  Government  contractor 
community.  The  Defense  Authorization  Acts  essentially 
preclude  the  Secretary  of  Defense  and  the  Military 
Departments  from  exercising  the  authority  to  implement 
management  and  internal  controls  over  the  amount  and  use  of 
IR&D  and  B&P  expenditures  by  Defense  contractors.  The  DoD 
will  have  to  rely  greatly  on  contractors'  business  sense  and 
responsibleness . 

Independent  Research  and  Development  and  Bid  and  Proposal 
Costs 


Contractor  IR&D  is  a  technical  effort  that  is  not 
sponsored  by,  or  required  in  performance  of,  a  contract  or 
grant  and  which  consists  of  projects  within  three  areas: 
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o  Basic  and  applied  research, 

o  Development,  and 

o  Systems  and  other  concept  formulation  studies. 

B&P  costs  are  the  expenses  incurred  in  preparing, 
submitting,  and  supporting  bids  and  proposals  on  potential 
Government  and  non-Government  contracts.  Such  IR&D  and  B&P 
costs  are  allowed  on  covered  contracts  to  the  extent  that 
the  costs  are  allocable,  reasonable,  and  not  otherwise 
unallowable  by  law  or  the  Federal  Acquisition 
Regulations  (FAR) . 

Before  1990,  Section  203  of  Public  Law  91-441  and  FAR 
31.205-18 (c)  required  any  company  receiving  IR&D  and  B&P 
payments  in  excess  of  $5.4  million  from  DoD  in  a  fiscal  year 
to  negotiate  an  advance  agreement  with  the  Government.  The 
purpose  of  this  advance  agreement  was  to  establish  a  ceiling 
for  allowability  of  IR&D  and  B&P  costs  for  the  following 
fiscal  year. 

Under  United  States  Code,  title  10,  the  Secretary  of 
Defense  was  required  to  prescribe  regulations  governing 
the  payment  of  IR&D  and  B&P  expenses  incurred  by 
contractors.  Accordingly,  management  and  internal  controls 
in  place  before  1990  generally  consisted  of  Government 
technical  experts  performing  an  on-site  technical  evaluation 
of  a  contractor's  IR&D  program;  the  contractor  and  the 
Government  negotiating  a  cost  ceiling  on  IR&D  and  B&P  based, 
in  part,  on  the  results  of  the  technical  review;  and 
contractors  recovering  negotiated  IR&D  and  B&P  costs  in 
overhead . 

The  National  Defense  Authorization  Act  for  Fiscal  Years 

1991  and  1992  repealed  Section  203,  Public  Law  91-441,  and 
increased  the  threshold  requirement  for  negotiating  an 
advance  agreement  with  the  Government  from  $5.4  million  to 
$7  million.  However,  a  limitation  on  the  amount  that  may  be 
paid  annually  for  FYs  1993  through  1995  to  major  contractors 
was  imposed. 

The  National  Defense  Authorization  Act  for  Fiscal  Years 

1992  and  1993  increased  the  threshold  to.  $10  million; 
included  the  use  of  a  specific  formula  by  which  the  maximum 
reimbursable  amount  may  be  determined;  and  changed  the 
allowability  ceiling  computations  for  contractor  fiscal 
years  beginning  after  September  30,  1992. 

The  National  Defense  Authorization  Act  for  Fiscal  Years 
1991  and  1992  broadened  the  definition  of  the  types  of 
research  and  development  projects  that  a  company  could 
consider  for  recovery  under  IR&D  from  those  projects  that 
have  a  potential  relationship  to  military  function  or 
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operation  to  those  projects  being  of  ''potential  interest"  to 
the  Department  of  Defense.  These  definitions  were  further 
broadened  for  Fiscal  Years  1992  and  1993.  The  new  Defense 
Federal  Acquisition  Regulations  Supplement  (DFARS) 
provisions  implementing  changes  from  the  Defense 
Authorization  Acts  are  far  less  stringent  than  former 
requirements . 

Specifically,  DFARS  231. 205-18 (c) (2)  now  provides  for 
seven  broad  categories  of  IR&D  and  B&P  projects  that  are  of 
potential  interest  to  DoD.  There  are  few,  if  any,  IR&D  and 
B&P  projects  that  a  Defense  contractor  would  propose  that 
would  fail  to  satisfy  at  least  one  of  the  following 
categories  of  projects: 

o  Enable  superior,  performance  of  future  United 
States  weapon  systems  and  components; 

o  Reduce  acquisition  costs  and  life-cycle  costs 
of  military  systems; 

o  Strengthen  the  U.S.  Defense  industrial  and 
technology  base; 

o  Enhance  the  U.S.  industrial  competitiveness; 

o  Promote  the  development  of  technologies 
identified  in  the  Defense-critical  technologies  plan  that  is 
submitted  to  Congress  annually  by  the  Secretaries  of  Defense 
and  Energy; 

o  Increase  the  development  of  technologies  useful 
for  both  the  private  commercial  sector  and  the  public 
sector;  or 


o  Develop  efficient  and  effective  technologies 
for  achieving  environmental  benefits. 

In  summary,  the  changes  legislated  by  the  Defense 
Authorization  Acts  have  served  to  expand  the  types  of 
research  and  development  projects  that  a  Defense  contractor 
could  consider  for  recovery  under  IR&D,  from  those  projects 
having  a  potential  relationship  to  military  function  or 
operation  to  those  projects  being  of  "potential  interest"  to 
the  Department  of  Defense.  The  Defense  Authorization  Acts 
have  also  eliminated  the  requirement  for  contractors  to 
submit  brochures  describing  IR&D  and  B&P  projects  for 
evaluation  and  "grading"  by  a  tri— Service  IR&D  technical 
committee  and  increased  the  cost  threshold  requirement 
for  an  advance  agreement  from  $5.4  million  to  $10  million. 
In  addition,  these  changes  have  modified  the  allowability 
ceiling  computations  for  contractor  fiscal  years  beginning 
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after  September  30,  1992,  and  allowed  for  the  "degree  of 
reasonableness"  to  prevail  in  contract  negotiations  for 
recovery  of  IR&D  and  B&P  costs. 

The  changes  legislated  by  the  Defense  Authorization 
Acts  have  also  precluded  the  Secretary  of  Defense  and 
Military  Departments  from  having  authority  to  implement 
management  and  internal  controls  over  the  amount  and  use  of 
IR&D  and  B&P  expenditures  by  Defense  contractors.  Further 
proliferation  of  redundant,  unvalidated  models  and 
simulations  could  be  an  unforeseen  by-product  of  the  new 
Government  policy. 

Conclusions 

Because  the  Secretary  of  Defense  and  the  Military 
Departments  are  not  authorized  to  implement  management  and 
internal  controls  over  the  amount  and  use  of  IR&D  and 
B&P  expenditures  by  contractors  and  the  impact  of  the 
recent  IR&D  and  B&P  policy  changes  is  not  yet  assessable, 
it  would  be  inappropriate  to  present  recommendations  for  any 
findings  developed  as  a  result  of  this  audit.  Pending 
further  evaluation  of  the  situation  concerning  contractor- 
owned  models  and  simulations  in  the  future,  we  believe  the 
best  course  is  for  the  DoD  to  put  its  own  house  in  order  by 
addressing  the  problems  identified  in  our  March  1993  report 
regarding  DoD-owned  models  and  simulations. 

Since  this  report  contains  no  recommendations,  written 
comments  were  not  required  and  none  were  received.  This 
report  does  not  claim  monetary  benefits. 

The  courtesies  extended  to  the  audit  staff  are 
appreciated.  If  you  have  any  questions  on  this  audit, 
please  contact  Mr.  Raymond  A.  Spencer  at  (703)  614-3995 
(DSN  224-3995)  or  Mr.  David  F.  Vincent  at  (703)  693-0355 
(DSN  223-0355) . 

Activities  visited  or  contacted  are  listed  in 
Enclosure  1.  The  planned  distribution  of  this  report  is 
listed  in  Enclosure  2. 


Robert  J .  Lieberman 
Assistant  Inspector  General 
for  Auditing 
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Rockwell  International,  Inc.,  El  Segundo,  CA 
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Westinghouse  Electric  Corporation,  Baltimore,  MD 
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Director,  Defense  Research  and  Engineering 

Department  of  the  Army 

Secretary  of  the  Army 

Inspector  General,  Department  of  the  Army 
Department  of  the  Navy 
Secretary  of  the  Navy 

Commandant  of  the  Marine  Corps  , 

Assistant  Secretary  of  the  Navy  (Financial  Management) 
Office  of  the  Chief  of  Naval  Operations 
Naval  Air  Systems  Command 

Department  of  the  Air  Force 

Secretary  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management 
and  Comptroller) 

Headquarters,  U.S.  Air  Force 

Air  Force  Studies  and  Analysis  Agency 

Wright  Laboratories 

other  DoD  Activities 

Director,  Defense  Information  Systems  Agency 
Director,  Defense  Logistics  Agency 
Director,  Defense  Nuclear  Agency 


ENCLOSURE  2 
Page  1  of  2 


REPORT  DISTRIBUTION 

(continued) 


Non-Defense  Activities 

Office  of  Management  and  Budget 

U.S.  General  Accounting  Office,  National  Security  and 
International  Affairs  Division,  Technical  Information 
Center 

Chairman  and  Ranking  Minority  Member  of  the  following 
Congressional  Committees  and  Subcommittees: 


Senate  Committee  on  Appropriations 
Senate  Subcommittee  on  Defense,  Committee  on 
Appr opr iat ions 

Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 
House  Committee  on  Appropriations 

House  Subcommittee  on  Defense,  Committee  on  Appropriations 

House  Committee  on  Armed  Services 

House  Committee  on  Government  Operations 

House  Subcommittee  on  Legislation  and  National  Security , 
Committee  on  Government  Operations 

Council  of  Defense  and  Space  Industries  Association 
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AUDIT  TEAM  MEMBERS 


Donald  E.  Reed 

Raymond  A .  Spencer 
David  F.  Vincent 
James  R.  Casey 
Thomas  N.  Wright 
Richard  L.  Collier 
Calvin  L.  Melvin 
Francis  M.  Ponti 

Henry  D .  Barton 

Dharam  V.  Jain 


Director,  Acquisition  Management 
Directorate 
Program  Director 
Project  Manager 
Team  Leader 
Team  Leader 
Auditor 
Auditor 

Program  Director,  Quantitative  Methods 
Division 

Operations  Research  Analyst, 
Quantitative  Methods  Division 
Operations  Research  Analyst, 
Quantitative  Methods  Division 


